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DETAILED ACTION 

1. Claims 1-41 are pending in this application and claims 1, 16-19 and 27 are 
canceled and claim 42 is newly added filed in amendment on 10/03/03 



Response to Arguments 

2. Applicant's arguments with respect to claims 12, 20, 24, 32 and 38-39 have been 
considered but are moot in view of the new ground(s) of rejection. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 2-3 and 28-31 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Atsushi et al. (A flexible content-based image retrieval system with combined 
scene description word, page 201-208, June 17-23. 1996, Multimedia Computing and 
System) in view of Barber etal. (US Patent No. 5,579,471). 

Regarding on claims 1,11 and 28, Atsushi teaches a method comprising: 
Initiating a search for images based on a at least one query keyword in an query 
(page 202, left column, lines 15-22); and 
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identifying, during the search, first images having associated keywords that 
match the query l<eyword and second images that contain low-level features similar to 
those of the first images (page 202, left column, lines 15-22); 

Atsushi does not explicitly teach ranking of the first and second images. Barber 
teaches "ranking the images according to the results" (col. 12, lines 30-33). This 
teaches the images are ranked according to their relevancy. Therefore, it would have 
been obvious to one ordinary skill in the art at the time of the invention was made to 
include the ranking method in Barber into Atsushi to allow the relevant images to be 
displayed in the ordering of match. 

Regarding on claims 3 and 31, Atsushi teaches presenting the first and second 
images (fig. 5). 

Regarding claim 29, Atsushi teaches a query handler comprises a natural 

language parser (page 202, left column, lines 15-22). 

Regarding on claim 30, Atsushi teaches the query handler comprises: 

A parser to parse text-based queries (page 202, left column, lines 15-22); and 

A concept hierarchy to define various categories of images (page 2, left column, 

lines 15-22). 

4. Claims 4-1 0, 20-23, 32-37 and 42 are rejected under 35 U.S.C. 1 03(a) as being 
unpatentable over Atsushi et al. (A flexible content-based image retrieval system with 
combined scene description word, page 201-208, June 17-23, 1996, Multimedia 
Computing and System) in view of He et al. (US Patent No. 6,557,042 B1 ). 
Regarding on claims 4 and 42, Atsushi teaches a method comprising: 
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Initating a search for images based on at least one query keyword in query (page 
202, left column, lines 15-22); 

Identifying, during a search, first images having associated keywords that match 
the query keyword and second images that contain low-level features similar to those of 
the first images (page 202, left column, lines 15-22); and 

presenting the first and second images to a user (page 104. left column, fig. 5); 

Atsushi does not explicitly teach monitoring feedback from the user as to which 
of the first and second images are relevant to the query. He teaches, "there are may be 
a relationship defined between a "marketing" group and an "engineering group, such 
that user feedback from a user in the marketing group also modifies the corresponding 
segment score in the engineering group. This modification may be the same (e.g., 
increase both scores by two for positive feedback), or different (e.g., for e positive 
feedback increase the segment score in the user's group by two and increase the 
segment score in the related group by one)" (col. 9, lines 56-64). This teaches the 
increasing the score by two when there are positive feedbacks from the user means 
annotating the segment by two of the marketing and engineering group. Therefore, it 
would have been obvious to one ordinary skill in the art at the time of the invention was 
made to include the user's feedback to modify the scores to annotate the marketing and 
engineering group to the multimedia object segment to allow the retrieval of the same 
segment that relevant to these marketing and engineering. 

Regarding on claims 5 and 32, Atsushi teaches a method comprising: 
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Imitating a search for images based on at least one query keyword in query (page 
202, left column, lines 15-22); 

Identifying, during a search, first images having associated keywords that match 
the query keyword and second images that contain low-level features similar to those of 
the first images (page 202, left column, lines 15-22); and 

presenting the first and second images to a user (page 104, left column, fig. 5) 
Atsushi does not explicitly teach receiving feedback from the user as to whether the first 
an second images are relevant to the query; and learning how the first and second 
images are identified based on the feedback from the user. On the other hand. He 
teaches, "there are may be a relationship defined between a "marketing" group and an 
"engineering group, such that user feedback from a user in the marketing group also 
modifies the corresponding segment score in the engineering group. This modification 
may be the same (e.g., increase both scores by two for positive feedback), or different 
(e.g., for e positive feedback increase the segment score in the user's group by two and 
increase the segment score in the related group by one)" (col. 9, lines 56-64). This 
teaches the increasing the score by two when there are positive feedbacks from the 
user means annotating the segment by two of the marketing and engineering group. 
Therefore, it would have been obvious to one ordinary skill in the art at the time of the 
invention was made to include the user's feedback to modify the scores to annotate the 
marketing and engineering group to the multimedia object segment to allow the retrieval 
of the same segment that relevant to these marketing and engineering. 



Application/Control Number: 09/702,292 Page 6 

Art Unit: 2172 

Regarding on claims 6 and 33. Atsushi teaches presenting the first and second 
images to a user (page 104, left column, fig. 5). He teaches, "there are may be a 
relationship defined between a "marketing" group and an "engineering group, such that 
user feedback from a user in the marketing group also modifies the corresponding 
segment score in the engineering group. This modification may be the same (e.g., 
increase both scores by two for positive feedback), or different (e.g., for e positive 
feedback increase the segment score in the user's group by two and increase the 
segment score in the related group by one)" (col. 9, lines 56-64). This teaches the 
increasing the score by two when there are positive feedbacks from the user means 
annotating the segment by two of the marketing and engineering group. Therefore, it 
would have been obvious to one ordinary skill in the art at the time of the invention was 
made to include the user's feedback to modify the scores to annotate the marketing and 
engineering group to the multimedia object segment to allow the retrieval of the same 
segment that relevant to these marketing and engineering. 

Regarding on claims 7 and 34, He teaches presenting the first and second 
images to a user (page 104. left column, fig. 5); 

receiving feedback from the user as to which of the first and second images are 
relevant to the query (col. 9, lines 49-64); and 

assigning a large weight to an association between the query keyword and the 
images deemed relevant by the user (col. 9, lines 49-64). 

Regarding on claims 8 and 35. Cullen teaches grouping the low-level features of 
the images deemed by relevant by the user (col. 4, lines 18-27). 
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Regarding on claims 9 and 36, He teaches presenting the first and second 
images to a user (page 104, left column, fig. 5); 

receiving feedback from the user identifying an example image as less or 
irrelevant to the query for refinement of the search (col. 9, lines 51-64); and 

assign a small weight to an association between the query keyword and the 
example image (col. 9, lines 59-64). 

Regarding on claims 10 and 37, Atsushi teaches identifying additional images 
with low-level features similar to those of the example image (fig. 5). 

Regarding on claims 20 and 23, Atsushi teaches the a method comprising: 

Presenting a result set of images that are returned that are returned from an 
image retrieval search of query having at least one keyword (col. 9. lines 40-50); 

Atsushi does not explicitly teach monitoring feedback from a user as to whether 
the image in the result set are relevant to the query; in an event that the user selects at 
least one image as being relevant to the query, associating the keyword in the query 
with the selected image to form a first keyword-image association and assigning a 
comparatively large weight to the first key-image association; and in an event that the 
user identifies an example image for refinement of the search, associating the keyword 
in the query with the example image to form a second keyword-image association and 
assigning a comparatively small weight to the second keyword-image association. He 
teaches, "there are may be a relationship defined between a "marketing" group and an 
"engineering group, such that user feedback from a user in the marketing group also 
modifies the corresponding segment score in the engineering group. This modification 
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may be the same (e.g., increase both scores by two for positive feedback), or different 
(e.g., for e positive feedback increase the segment score in the user's group by two and 
Increase the segment score in the related group by one)" (col. 9, lines 56-64). This 
teaches the increasing the score by two when there are positive feedbacks from the 
user means annotating the segment by two of the marketing and engineering group. 
Therefore, it would have been obvious to one ordinary skill in the art at the time of the 
invention was made to include the user's feedback to modify the scores to annotate the 
marketing and engineering group to the multimedia object segment to allow the retrieval 
of the same segment that relevant to these marketing and engineering. 

Regarding on claim 21, Atsushi teaches conducting both content-based image 
retrieval and semantic-based image retrieval (page 204, right column, lines 25-31). 

Regarding on claim 22, He teaches presenting the result set of images In a user 
interface, the user interface facilitating the user feedback by allowing the user to 
indicate which images are more relevance and which images are less relevant (col. 9, 
lines 51-56). 

5. Claims 12-15 and 39-41 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Barber (US. Patent No. 5,579,471) in view of He et al. (US. Patent 
No. 6,557.042). 

Regarding on claims 12, 15, 39 and 41, Barber teaches a method comprising: 
Permitting entry of both keyword-based queries (keyword) and content-based 
queries (image query) (col. 3, lines 23-38); 
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Finding images using both semantic-based image retrieval and low-level feature- 
based image retrieval (col. 5, lines 30-42); 

Presenting the images to a user so that the user can indicate whether the images 
are relevant (col. 5, lines 30-42); and 

Barber does not explicitly teach conducting semantic-based relevance feedback 
and low-level feature-based relevance feedback in an integrated fashion. However, He 
teaches, "there are may be a relationship defined between a "marketing" group and an 
"engineering group, such that user feedback from a user in the marketing group also 
modifies the corresponding segment score in the engineering group. This modification 
may be the same (e.g.. increase both scores by two for positive feedback), or different 
(e.g., for e positive feedback increase the segment score in the user's group by two and 
increase the segment score in the related group by one)" (col. 9, lines 56-64). This 
teaches the increasing the score by two when there are positive feedbacks from the 
user means annotating the segment by two of the marketing and engineering group. 
Therefore, it would have been obvious to one ordinary skill in the art at the time of the 
invention was made to include the user's feedback to modify the scores to annotate the 
marketing and engineering group to the multimedia object segment to allow the retrieval 
of the same segment that relevant to these marketing and engineering. 

Regarding on claims 13 and 40, Barber teaches ranking the images (col. 12, 
lines 30-33). 

Regarding on claim 14, Barber teaches using images indicated as being relevant 
to find additional images (col. 14, lines 46-47). 
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6. Claim 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over He et al. 
(US. Patent No. 5,899,999). 

Regarding on claim 38, He teaches a database structure stored on one or more 
computer readable media comprising: 

Multiple image files (col. 3, lines 35-45); 

Multiple keywords (col. 4, lines 9-1 1 ); and 

A sematic network to associate the keywords with the image files, the semantic 
network defining individual keyword-image links that associate a particular keyword with 
a particular image file, each keyword-image link having a weight indicative how relevant 
the particular key word is to the particular image file. However, He teaches, "there are 
may be a relationship defined between a "marketing" group and an "engineering group, 
such that user feedback from a user in the marketing group also modifies the 
corresponding segment score in the engineering group. This modification may be the 
same (e.g., increase both scores by two for positive feedback), or different (e.g., for e 
positive feedback increase the segment score in the user's group by two and increase 
the segment score in the related group by one)" (col. 9, lines 56-64). This teaches the 
association between the segments and marketing and engineering group. Therefore, it 
would have been obvious to one ordinary skill in the art at the time of the invention was 
made to include the user's feedback to modify the scores to annotate the marketing and 
engineering group to the multimedia object segment to allow the retrieval of the same 
segment that relevant to these marketing and engineering. 
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7. Claims 24-25 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Nojima (US. Patent No. 6,442,438) in view of Narayanaswami et al. (US. Patent No. 
6.504,571). 

Regarding on claim 24, Nojima teaches a method comprising: 

Computing, for each category, a representative feature vectors of a set of 
existing images with the category (col. 4, lines 30-41 ); 

Determining a set of representative keywords that are associated with the 
existing images in each category (col. 5, lines 50-57); 

Comparing, for each new image, the low-level feature vectors of the new image 
to the representative feature vectors of the existing images in each category to identify a 
closet matching category (col. 1, lines 19-37); and 

Nojima does not explicitly teach labeling the new image with the set of 
representative keywords associated the closet matching category; however, 
Narayanaswami teaches, "the system also includes an image annotation module 220, 
operatively connected between the user interface/display 202 and the images from the 
image database 216, which allows a user to retrieve digital images with additional 
parameters or keywords such as names and descriptions of objects in the images" (col. 

8, lines 52-58). This teaches the annotation module annotates the keywords to the 
images. Therefore, it would have been obvious to one ordinary skill in the art at the 
time of the invention was made to combine the teaching Nojima and Narayanaswami 
because utilizing the annotation module to annotate the keyword to the images would 
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allow the images to be retrieved by other keywords that are more descriptive to the 
images. 

Regarding on claim 25, Narayanaswami teaches using feedback to selectively 
add and/or remove (annotating means adding) keywords from the new image (col. 8, 
lines 52-58). 

8. Claim 26 is rejected under 35 U.S.C. 103(a) as being unpatentable over Nojima 
(US. Patent No. 6,442,438) in view of Narayanaswami et al. (US. Patent No. 
6,504,571)andfutherin view of Kopec etal. (US. Patent No. 5,594,809). 

Regarding on claim 26, Both Nojima and Narayanaswami teach placing the 
labeled new images into a holding category (col. 8, lines 53-57). 

Both Nojima do not explicitly teach evaluating the labeled new images in the 
holding category to determine if any of the keywords associated with the labeled new 
image match the representative keywords from each category; and assigning the 
labeled new image to the category that best matches the keywords associated with the 
labeled with the labeled new image. 

Spec teaches evaluating the labeled new images in the holding category to 
determine if any of the keywords associated with the labeled new image match the 
representative keywords from each category (col. 7, lines 63-65); and assigning (input 
keyword image) the labeled new image to the category that best matches the keywords 
associated with the labeled with the labeled new image (col. 7, lines 64-65). 

Therefore, it would have been obvious to one ordinary skill in the art at the time 
of the invention was made to employ the teaching of Kopec into Nojima and 
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Naraynaswami in order to match the image with the keyword to allow the user search 
for these images with the same keyword. 

9. Claim 29-30 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Nojima (US. Patent No. 6,442,438) in view of Narayanaswami et al. (US. Patent No. 
6,504,571 ) and further in view of Stuckey et al. (US. Patent No. 5,721 ,938). 

Regarding on claim 29, both Nojima and Narayaswami do not explicitly teach the 
query handler comprises a natural language parser; however, Stuckey teaches, 
"present program provides a natural text parser, which may be used for all natural 
languages" (col. 3, lines 22-24). This teaches the query is inputted by the natural 
language. Therefore, it would have been obvious to one ordinary skill in the art at the 
time of the invention was made to modify the teaching of Stuckey into Nojima and 
Narayanaswami because utilizing the natural language parser would allow the user to 
input the natural language to search and to provide a user-friendly system. 

Regarding on claim 30, Stuckey teaches the query handler comprises: 
A parser to parse text-based queries (col. 3, lines 22-24); and 
A concept hierarchy to define various categories of images (col. 12, lines 22-25). 

Conclusion 

1 0. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Baoquoc N. To whose telephone number is (703) 305- 
1949 or via e-mail BaoquocN.To@uspto.gov. The examiner can normally be reached 
on Monday-Friday: 8:00 AM -4:30 PM, EST. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John Breene can be reached at (703) 305-9790. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
3900. 

Any response to this action should be mailed to: 



The fax numbers for the organization where this application or proceeding is 
assigned are as follow: 

(703) 872-9306 [Official Communication] 

Hand-delivered responses should be brought to: 
Crystal Park II 
2121 Crystal Drive 
Arlington. VA 22202 
Fourth Floor (Receptionist). 



Commissioner of Patents and Trademarks 



Washington, D.C. 20231. 



Baoquoc N. To 
Dec 23, 2003 




